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ABSTRACT 

This review attempts to describe the current state of 
Educational Testing Service (ETS) activity in innovative instruction 
and assessment at the classroom level. Project reports are preceded 
by summaries of conclusions drawn from recent research and current 
issues under examination. Research now supports much that was 
previously hypothesized, confirming that students can adapt to' new 
expectations and new ways of learning and afsessment. Research also 
suggests that naturalistic assessment methods are appropriate for the 
primary level and that curriculum materials from "real life" are 
effective in promoting learning. Educational technology can play an 
important role in instruction, and research is supporting its use in 
various media. Current issues focus on: (1) assessment methodology; 
(2) psychometric concerns; (3) cognitive issues; (A) equitable 
performance tasks for all students; and (5) teacher preparation and 
development. One table summarizes ongoing projects. Individual 
project summaries are given for AO ETS projects. (SLD) 
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Assessment and Instruction at the Classroom Level: 
Current and Recent ETS Projects 



Introduction 

This review was prepared as background material for the Consortium on Educational 
Process and Technology, a collaborative effort among ETS, the David Sarnoff Research Center, 
and the WUliamstown Education Foundation. The goal of the consortium is to combine 
technological and educational planning to create learning environments that will motivate 
students, help them achieve high levels of learning, and provide equity to all learners. 

The review is an attempt to describe the current state of ETS activity in innovative 
instruction and assessment at the classroom level. Most of the projects included in the review 
are concerned with what has recently come to be known as "instructional assessment" — 
assessment that is designed to inform instruction. AH of the projects contribute toward the 
development of the instructional assessment concept, although a few are designed to focus more 
narrowly on either assessment or instruction. AH of the projects involve one or both of the 
major innovations in classroom instruction — technology and performance assessment. 

The format of the review is designed to facilitate updating as new projects are initiated 
and as new conclusions emerge from projects currently in progress. 

■ What We Have Learned summarizes the conclusions drawn from research conducted to 
date. Although there are no surprise.^ here, it is important to note that we now have 
evidence to support what were merely hypotheses a few years ago. 

■ Summary of Current Issues summarizes the issues under examination in current 
projects. Many of the issues are being addressed in more than one project at different 
grade levels, in several curriculum areas, and from a variety of perspectives. 

■ Issues Being Addressed in Current Projects lists the questions and issues that are of 
current concern and specifies the projects in which they are being addressed. 

■ Assessment and Instruction at the Classroom Level: Overview is a chart that presents 
basic information about all the projects reviewed. The chart includes the status of the 
project (current or completed), the educational level and subject matter targeted by the 
project, the focus of the project (assessment, instruction, or instructional assessment), the 
type of technology used, whether the project involves performance assessment, anu 
w'iiether the project includes a teacher training component. In order to keep similar 
projects near each other, projects are grouped by educational level and content area. 

Information on projects was assembled primarily from existing sources: from project reports, 
abstracts, and proposals; from information gathered recently in the preparation of updates for 
the Research Reporter, and from information gathered for the summary of portfolio and work- 
sample projects prepared last year. When no current information was available in print, project 
directors were interviewed by telephone. 
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What We Have Learned 



Students at all educational levels can adapt to new expectations and new ways of 
learning, such as becoming aware of and active in the learning process, taking more 
responsibility for their own learning, and self-evaluation. 

Evidence : Rhode Island Literacy Portfolio Project (Fowles) • Elementary 
R & D in Mathematics (Lesh) - Middle 
Middle School Science (Gong) - Middle 
Arts PROPEL (Gitomer) - KiSfPe, Secondary 
Computer Integration in Math (Morris) - Secondary 

At the primary level, naturalistic assessment methods ("documentation") are appropriate 
and effective in reading instruction -nd may be applicable to other subject areas. 

Evidence : Classroom-Based Assessment (Chittenden) - Elementary 

Curriculum materials drawn from "real-life" situations and problems are effective in 
engaging students' attention, in promoting learning at both the factual and conceptual 
levels, and in helping students to acquire basic thinking and problem-solving skills. 

Evidence : Interactive Videodisc Applications to Math (Bejar) - Elementary 
R & D in Math (Lesh) - Middle 
Middle School Science fGo/jgj - Middle 

Integration of the computer into writing instruction improves the quality of students' 
writing by making it easy for students to revise drafts and by creating products that 
increase students' pride in their own writing and their interest in reading the writing of 
others. 

Evidence : Maryland Education Project (Fowles & Wilder) - Elementary, Middle 

Artificial intelligence can be used to score constructed responses in some subjects, 
producing partial credit scores where appropriate and diagnostic feedback about errors 
in either the answer or the process used to arrive at it. 

Evidence : Using AI to Score Constructed Responses in Algebra CBen/ieffj -Secondary 
AP Computer Science Practice System CBe/Jneffj -Secondary 

Systems thinking with simulation modeling enhances students' problem-solving skills and 
their conceptual understanding of subject matter. 

Evidence : STACI- (Mandinach) - W§M' Secondary 
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Videodisc-based lessons featuring an electronic student response system are effective in 
promoting learning for students who bring little previous knowledge and/or low self- 
confidence to the subject. 

Evidence : Evaluation of a Mathematics Videodisc Project (Moe) - Middle 

Two design principles that help to promote equity in performance assessments are 

1) keeping the assessment tasks relatively short (limited to two class periods, for 
example) to reduce the opportunities for some students to take advantage of 
resources available to them but not to others {e.g., older siblings, interested 
parents, materials in the home); 

2) allowing students some choice in response modalities so that no students are 
penalized by having to use a form of communication that is particularly difficult 
for them. 

Evidence : Pacific Telesis program (Jorgensen) - Elementary, Middle 

Teachers can adapt to the new role of facilitator or coach required by the shift to 
technology-based and/or performance-based instruction and assessment; but the change 
in the educat;3nal model is dramatic, and teachers need training, guidance, 
organi2:ational support, and adequate planning time in order to implement it successfully. 

Evidence : Classroom-Based Assessment (Chittenden) - Elementary 
Middle School Science ( Gong) - Middle 
Arts PROPEL (Gitomer) - Middle, Secondary 
BJA<^*^ (Mandinach) - Middle, Secondary 

Two critical factors in the successful implementation of technology-based programs in 
public schools are 

1) a close match between the technology and the existing curriculum and 
instructional patterns; 

2) strong and continuing administrative leadership and support. 

The changes required in teachers' roles and responsibilities can be a barrier to successful 
implementation, but this can be overcome by strong, committed leadership that fosters a 
cUmate for change. 



Evidence : Implementation of Technology in Schools (Stecher & Wilder) - Ail levels 



Videotapes of demonstration classrooms are powerful communication tools that help 
teachers, administrators, and parents understand the purposes and the etfectiveness of 
classroom innovations such as naturalistic assessment, portfolios, and instruction that 
nurtures the development of higher order thinking skills and metacognitive processes. 

Evidence : Videotape Demo of Naturalistic Assessment (Chittenden) ■ Elementary 
Pacific Telesis program (Jorgensen) - Elementary, Middle 
Reflections of Learning (Jorgensen) - Elementary, Middle, Secondary 

Video portfolios including tapes of teachers' classrooms can be used as a means of 
assessing accomplished teaching; the process of preparing portfolios and the process of 
learning to score them contribute to the professional development of ♦he participants and 
have beneficial effects on participants' classroom practice. 

Evidence : Video Portfolio Assessment (J. Frederiksen) - Secondary 



Summary of Current Issues 

Methodology 

Creation of new or improved performance measures and scoring systems. 

Extension of successful methods from one content area into others. 

Developing a way to aggregate information across automatically scored items to detect 
and describe Lnstructionally meaningful patterns in the scores and diagnostic information 
produced by the system. 

Providing evidence for enhanced student learning in forms that can be communicated to 
administrators, parents, and others concerned with accountability. 

fniK!W«cilttp'^agibt«lar^^ i;«$t sc^te^'t>etfotm^ooe me$smeR{^ and teacher 

UseMjBtafonmoace a&sessments to tcfeati^' pfed md talented students. 

Psychometric Concerns 

Ensuring reliability of performance-based measures. 

Establishing validity of performance-based measures. 

Exploring the role of standardization in performance-based measures. 

Examining the ways in which differing degrees of structure in open-ended questions 
affect the effectiveness of an automated scoring system. 

Testing the applicability of new, cognitively driven measurement models to interactive 
assessments using AI. 

Cognitive Issues 

Determining whether constructed-response item types in algebra measure a different 
dimension from that measured by multiple-choice items. 

Determining what mathematical skills are prerequisite to students' effective use of 
simulation modeling techniques. 

Det<",rmining the effects of maturity and cognitive development on students' ability to 
benefit from the systems thinking approach. 
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Learning what alternative assessments reveal about students' understanding at the 
conceptual level; learning what mental models they use to understand and explain 
scientific phenomena and how different forms of instruction contribute to development 
of different kinds of models. 

Understanding the cumulative effect on performance of students' working on a sequence 
of performance tasks or a series of systems thinking courses. 

Finding evidence of generalizability and transfer of thinking skills developed through 
performance- or technology-based instruction. 

Map»»fSPiSB?^^fBJipfrp^^ pM^t:iamlcitWmm Eripsh or a 

Equity 

Ensuring equity by designing performance tasks that are engaging and revealing for all 
students. 

Designing performance tasks and instructional models that will increase participation in 
math and science by members of disadvantaged groups and/or by groups in which such 
participation is traditionally low. 

BxaiiJioatioft of dfc«noa&^among ei^mlt^^gesder^^aad gra«le4evrel groups on. three forms 
of-^ssesmitftfttfst^tdar^^test^ ^^<^ permit 

EiteimtessDf a test preparation program for .deaf students, tising interactive videodisc 

Teacher Preparation and Development 

Development of effective techniques for training teachers in the use of performance- 
based and/or computer-based iastruction and assessment. 

Development of course-specific guidelines to help teachers conduct classroom activities 
and select instructional tasks. 

Determinmg the 'teacber training and curriculum ^^elopment activities that should 
accorajpany .the establishment of high techitology classrooms. 

Training and assisting teachers in the design of materials for performance-based 
instruction and assessment. 
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iSaiiaSh^^^f ISieli^^ used by'choral mtiskteacheiriS^to (^mmunicate 

Development of performance-based standards and assessment procedures for teachers, 
kiiot^lg&^^a'^m^ ollmpiemeatkMaitaim^ In sdeace and 

Classroom Practice 

Discovering ways to facilitate development of an adaptive teaching/learning 
environment. 

Examining the effects of portfolios and other performance measures on classroom 
practice. 

Comparing the effectiveness of various strategies for integrating technology-based 
activities into the curriculum. 

School Organization and Administration 

Documentin<; ways in which school organization and climate can promote and support 
the shift to performance-based or technology-based instruction in the classroom. 

Usefulness for accountability and parent-communication purposes of information gained 
through performance-based measures. 

Benefits and drawbacks of a teacher-centered versus an instrument-centered assessment 
program. 
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Issues Being Addressed in Current Projects 

Methodolog y 

■ Applicability of naturalistic methods developed for primary-level reading to other 
primary-level subjects. 

Projects : Yoimg Children's Science Learning (Chittenden) - elementary 

' "Wife Mmocit?!^^ 

m Design of curriculum-embedded performance assessments in reading, writing, math, and 
science at all grade levels. 

Projects. ^t^t^^^^-^^^^^^^^t '^f S^^^^^^^-^^^ ^MT^ 

California Ass^ment"Projecrf5/tei«^oW)"- elementary-secondary 
Pacesetter i^^^H - secondary 

■ Design of criteria for organic portfolios in reading, writuig, math, and science, to be used 
statewide. 

Project : California Assessment Project (Sheingold) • elementary-secondary 

■ Development of common goals, standards, and documentation strategies for performance 
assessment activities that will yield aggregatable data, in math and science in grades 3-6. 

Project : Authentic Assessment our Multiple Users (Jorgensen) • elementary 

■ What qualities of performance should be assessed, and how should they be presented? 

Project : Performance Assessment in Middle School Science (Frederiksen) - middle 

■ Extension of AI scoring systems to other classes of content and other difficulty levels 

Projects : AI for Constructed Responses in Alpibra (Bennett) • secondary 
AP Computer Science Practice System (Bennett) - secondary 

■ Refinement of algebra AI scoring rubric to reduce remaining discrepancies between 
human and automatic scoring. 

Project : AI for Constructed Responses in Algebra (Bennett) • secondary 

■ The characteristics of effective questions to assess complex understanding in science. 

Project : Assessment of Complex Understanding in Science Education (Gitomer) - 
secondary 
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■ latfeEi«l^tiOBs!bB^ •afed%'statfH3a«le;ed test^scaresi |jei:&rmaiK» a^assments^ and teacher 

Project : AI^iti?e,^jpedi$i;Bd8catfe^ A^^esstbeftt of lM>$tn ^iinoriQf Students 

Project : /y&f^alyel^Si^^iEiiD^ Urban 
Psychometric Concerns 

■ Ways in which differing degrees of structure in open-ended questions affect the 
effectiveness of an automatic scoring system. 

Project : AI for Constructed Responses in Algebra (Bennett) ■ secondary 

■ Design of criteria for grading portfolios and other forms of performance assessment, to 
ensure reliability. 

Projects : California Assessment Project (Sheingold) • elementary-secondary 
R & D in Math (Lesh) • middle 

Performance Assessment in Middle School Science (Frederiksen) - middle 
issues and -Technical F!m;edur«» in Rating JPortfalios (M^ord S: Miskvy) - 

■ Validity of performance assessments and portfolio-based assessments. 

Projects : South Brunswick Portfolio Project {Chittenden) - elementary 

California Assessment Project (Sheingold) - elementary-secondaiy 
hrn^ and Technical Broc«dor«s ^'Rating Portfolios (Myford A Mistevy) • 

■ Necessity for standardizing and controlling delivery of assessment tasks, versus using 
alternative tasks designed by teachers and/or embedded within a curriculum. 

Project : Performance Assessment in Middle S:hool Science (Frederiksen) - middle 

■ Aggregation of information aaoss automatically scored items to detect and describe 
instructionally meaningful patterns in scores and diagnostic information. 

Project : AI to Score Constructed Responses in Algebra (Bennett) - secondary 

■ Applicability of new, cognitively driven measurement models to interactive assessments 
using AI. 

Project : AI to Score Constructed Responses in Algebra (Bennett) - secondary 
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Do constructed-response item types in algebra measure a different dimension from that 
measured by multiple-choice items? 

Project : AI to Score Constructed Responses in Algebra (Bennett) - secondary 

Effectiveness of "authentic" mathematics assessments in increasing students' 
understanding of and ability to use mathematical principles. 

Project : R & D in Math (Lesh) • middle 

Development of strategies and evaluation criteria that assess students' knowledge bases 
in science and also reflect the principles of scientific thinking. 

Project : SEPIA (Gitomer) - middle 

Assessment of Complex Understanding in Science Education (Gitomer) ■ 
secondary 

What alternative assessments reveal about students' understanding. 
Project : SEPIA (Gitomer) • middle 

Understanding of the mental models students use to understand and explain scientific 
phenomena and of the ways in which different forms of instruction contribute to the 
development of different kinds of models. 

Proje ct: Middle School Science (Gong) • middle 

Cumulative effect on student performance of engaging in a sequence of assessment tasks 
throughout the duration of a course. 

Project : Performance Assessment in Middle School Science (Frederiksen) - middle 

Cumulative effect on analytic reasoning skills of progressing through several years of 
systems thinking courses. 

Project : STAQ** (Mandinach) • middle, secondary 

Impact of the systems approach on students' acquisition of content knowledge. 

Project : STACI** (Mandinach) - secondary 

Determining what mathematical skills are prerequisite to the effective use of simulation 
modeling techniques. 

Project : STAO''^ (Mandinach) ■ imiMe, secondary 
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■ Effects of maturity and cognitive development on students' ability to benefit from the 
systems approach. 

Project : ^E'^' (Mandinach) • ^IH, secondary 

« Mal^^l ^^eaf ^j3jMtti*,r^^lB£ perfonnajjce m6 |^*r^ far writtm En0kh or a 

« IWI^^^ olthe dljffiK«Me& (J«af students hstm 'aciJi^us^ tb.em«anibg of writte» 

middki 

Equity 

■ Ensuring equity by designing performance tasks that are engaging and revealing for all 
students. 

Project : California Assessment Project (Sheingold) - elementary-secondary 

■ Design of performance assessments targeted toward groups in which math participation 
is currently low. 

Project : R & D in Math (Lesh) - middle 

■ Determining what instructional models wiU develop skill in scientific inquiry in less 
advantaged populations. 

Project : Performance Assessment in Middle School Science (Frederiksen) - middle 

■ Bx)ym8ti<m"'<3f <M«r«ace&"'amc«ig ^tixaicj pntder^ srasS^gra^ifrievil ]p:oups on fhree forms 
of assessttte»t: staudatdized testSa^ ptsaformaiice asfes^ents, atid teacher aad parent 
ot»«mtions. 

Project : Alternative Special Education Assessment of Uiban Mmority Students' 

■ Use of i^ormance asse$$ttieatst to l^tilgf (Mdt^.ndth «?(ce^jional ^ibiKiy in science 

Project : Aiteriiative Assessment of ^en.<» and Math for Barly Childhood Urban 
Mmority Students f/wor) elementally 
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Teacher Preparation and Development 

■ Appropriate and effective techniques for training teachers in the use of performance- 
and/or computer-based instruction. 

Projects : Documenting Young Children's Snence Learning (Chittenden) ■ 
elementary 

Videotapes of Naturalistic Assessment Methods (Chittenden) ■ elementary 
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SEPIA (Gitomer) - middle 

Middle School Science (Gong) • middle 

Performance Assessment in Middle School Science (Frederiksen) ■ middle 
Assessment of Complex Understanding in Science Education ( Gitomer) ■ 
secondary 

STACl'* (Mandinach) • tniMe, secondary 
Pacesetter (l^jmnwl) • secondary 



Training and assisting teachers in the design of performance-based instruction and 
assessment. 

Projects : Alt^ati^e A$s4ssffl^t of Sdeso^ mi for^Basrly Chfldliood Urb^tn 

Gasacwm-^^ 1!>dit»XB«EitatiOB of Iltfe^tQf aM'Sraenc© Devdopment in 

Urfcait Mittodly C^ildt«ii (Jmes) « eiementafj? 
R «& D in Math' (Lesh) ■ middle 

Developing guidelines to help teachers conduct classroom activities and select 
instructional tasks. 

Project : SEPIA (Gitomer) ■ middle 

How can teachers use the knowledge gained through performance assessments to inform 
further instruction? 



Projects : SEPIA (Gitomer) • middle 

Middle School Science (Gong) - middle 
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■ Development of a model of the scoring process used to assess video portfolios of 
classroom teachers. 



Project : Video Portfolio Assessment (J. Fr^deriksen) • secondary 

■ Examination of the role of video clubs in the development of video portfolios for teacher 
assessment: What are the effects of the use of exemplar tapes? How can clubs construct 
libraries of exemplar tapes from their own districts? 

Project : Video Portfolio Assessment (J. Frederiksen) - secondary 

» Appropriateness and effectiveness of the standards and assessment practices advocated 
by the National Board for Professional Teaching Standards in assessing teachers and 
contributing to their professional development. 

Project : Implementation of a Staff Development Model in the South Brunswick 
Public Schools (Chittenden) ■ elementary 

Classroom Practice 

■ Ways to facilitate development of an adaptive teaching/learning environment. 

Project : Middle School Science (Gong) - middle 

■ Effects of portfolios or other performance-based techniques on classroom practice. 

Projects : South Brunswick (Chittenden) ■ elementary 

Performance Assessment in Middle School Science (Frederiksen) - middle 

■ Effectiveness of a portfolio-based approach in fostering a conceptual-change model of 
teaching. 

Project : SEPIA (Gitomer) - middle 
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■ Effects of the use of various strategies for integrating ted^o^iB^ simulation 
modeling activities into the curriculum. 

Project : ^^Hf' (Mandimch) • ^^^^ secondary 

■ How does participation in a video club affect teaching practice? 

Project : Video Portfolio Assessment (J. Frederiksen) - secondary 
School Organization and Administration 

■ Ways in which school organization and climate can promote and support the shift to 
performance-based or technology-based instruction at the classroom level. 

Project : Middle School Science (Gong) - middle 

* To what extent can evidence gained by naturalistic methods replace test scores as critical 
data? 

Projects : Formative Evaluation of Naturalistic Assessment Methods for Beginning 
Reading (Chittenden) • elementary 
SEPIA (Gitomer) • middle 

■ What are the benefits and drawbacks of a teacher-centered versus an instrument- 
centered assessment program? 

Project : Formative Evaluation of Naturalistic Assessment Methods for Beginning 
Reading (Chittenden) - elementary 

■ What are die s^atn»wide ^imjpUcatioas of the 'm&M altetnative assessmei^t pr ocedttres? 

Project : Oassroom-^as^sd t>oaimefitatio» p| liiteracy^aud Sd^ce 'Development in 
Uidjaa Mittodtty OMt^klSmesj^Stmor^ - ekmentaiy 
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Implementation of a Staff Development Model in the South Brunswick Public Schools 



Project Director Ted Chittenden 
Project Status Current 
Educational Level Elementary 
Subject Matter Many 
Focus Assessment 
Type of Technology Used Videotapes 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 



The purpose of the project is to develop models for professional staff development that 
use the standards and assessment measures advocated by the National Board for 
Professional Teaching Standards. A core group of teachers will initiate the process of 
using the standards and exploring their value as a framework for assessing the work of 
teachers in the classroom. Based on this experience, a model of staff development will 
b ^ clarified and tried out throughout the district. The use of videotapes will follow the 
pattern established in the Video Portfolio Assessment Project. 



Questions/Issues Addressed 

How effective are the standards as a framework for assessing teaching practice? 

How effective are the standards in contributing to individual teachers' professional 
development? 

Questions/Issues Remaining 

N/A 
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Classroom-Based Assessment in Early Childhood Education 

Project Director Ted Chittenden 
Project Status Completed 
Educational Level Elementary 
Subject Matter Reading 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 

Description 

Development of "naturalistic" assessment methods to document the developing reading 
skills of K-3 students. 



Questions/Issues Addressed 

Can the outcomes of research on children's emergent reading be converted into 

st rategies that classroom teachers can use in place of traditional tests to monitor and 

?.ssess the development of children's reading ability? Yes 

Will the use of such strategies and instruments contribute to more effective reading 
instruction? Yes 



Questions/Issues Remaining 

To what extent can evidence yielded by naturalistic methods replace test scores as critical 
data? 

What are the benefits and drawbacks of a teacher-centered versus an instrument- 
centered assessment program? 

Both questions are being addressed in other studies, i.e., Formative Evaluation of 
Naturalistic Methods for Assessing Beginning Reading and South Brunswick Portfolio 
Project. 
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Videotape Oeu;. nstration of Naturalistic Assessment Methods 



Project Director Ted Chittenden 
Project Status Current 
Educational Level Elementary 
Subject Matter Reading 
Focus Instructional Assessment 
Type of Technology Used Videotape 
Performance Assessment? Yes 
Teacher Training Component? Yes 

Description 

Development of a series of videotapes and supporting materials to illustrate the use of 
systematic, naturalistic methods for documenting young children's progress in learning to 
read. 



Questions/Issues Addressed 

Can videotapes be used effectively to help teachers learn how to use naturalistic 
assessment strategies? Yes 

Can videotapes be used effectively to illustrate the method to administrators considering 
adopting naturalistic methods and to parents whose children are in such classrooms? 
Yes 



Questions/Issues Remaining 

N/A 
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Formative Evaluation of Naturalistic Methods for Assessing Beginning Reading 

Project Director Ted Chittenden 
Project Statiis Current 
Educational Level Elementary 
Subject Matter Reading 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? No 



Description 

Evaluation of the use of alternative assessment methods and their impact on the school 
organization. 

Questions/Issues Addressed 

To what extent can evidence yielded by naturalistic methods replace test scores as critical 
data? 

What are the benefits and drawbacks of a teacher-centered versus an instrument- 
centered assessment program? 

Questions/Issues Remaining 



N/A 



South Brunswick Portfolio Project 

Project Director Ted Chittenden 
Project Status Cuaent 
Educational Level Elementary 
Subject Matter Reading 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 

Description 

Development of a portfolio assessment system for a "developmentally appropriate" 
reading program in grades K-2. 

Questions/Issues Addressed 

Can such a system by designed? Yes 

Can appropriate instruments be developed to give structure and consistency to the 
portfolio assessment system? Yes 

Can r rating system be designed to ensure the credibility of the portfolio assessment 
system beyond individual classrooms? A holistic rating system and literacy scale have 
been designed. 

How does the use of the portfolio affect classroom practice? 

How reliable and valid is the literacy scale used for rating the contents of the portfolios? 

Questions/Issues Remaining 

N/A 
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Alternative Special Education Assessment of Urban Minority Students 

Project Director Jacqueline Jones 
Project Status Completed 
Educational Level Elementary 
Subject Matter Reading, Writing 
Focus Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

The study examined the interrelationships among three methods of gathering evidence 
about the abilities of a group of urban minority special education students. The three 
forms of evidence were standardized test scores, newly developed performance-based 
instruments, and teacher and parent observations. The study also examined the mean 
differences among ethnic, gender, and grade level groups on the three assessment types. 
The project director met weekly with groups of special education teachers and parents of 
children enrolled in the special education programs, to assure competence and 
consistency in the use of the portfolio and observation ratings. The project took place in 
an elementary school in the South Bronx, New York City. The students in the study 
were 130 Hispanic and African American students, predominately classified as learning 
disabled. 



Questions/Issues Addressed 

Interrelationships among standardized test scores, performance-based assessments, and 
teacher and parent observations as measures of student progress and ability. 

Differences among ethnic, gender, and grade level groups on the three forms of 
assessment. 

Questions/Issues Remaining 

Can the understandings gained be generalized to other groups of students and to other 
subject matter? Being addressed in "Alternative Assessment of Science and 
Mathematics for Early Childhood Urban Minority Students," a follow-up to this project. 



Evaluation of NYC District 27 Alternative Assessments 

Project Director Jacqueline Jones 
Project Status Completed 
Educational Level Elementary 

Subject Matter Linguistic, logical-mathematical, interpersonal, intrapersonal, and spatial 
abilities 

Focus Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 

Description 

This project provided consulting services to New York City Public School District 27 in 
connection with their design of alternative assessments to identify gifted and talented 
students. The project director consulted with the Director of Gifted and Talented 
Programs and with staff involved in the administration, scoring, and interpretation of the 
assessment instruments. A critical analysis of the criteria and scoring dimensions was 
conducted. 



Questions/Issues Addressed 

Use of performance assessments to identify gifted and talented students. 
Identification of appropriate criteria. 
Development of scoring standards. 

Questions/Issues Remaining 



None 



Teachers' Subject Matter Representations in Elementary School Music 



Project Director Anne Reynolds 
Project Status Current 
Educational Level Elementary 
Subject Matter Music 
Focus Instruction 
Type of Technology Used None 
Performance Assessment? No 
Teacher Training Component? No 



Description 



To communicate information in meaningful ways, teachers use representations such as 
diagrams, verbal metaphors, demonstrations, illustrations, and tone of voice. When 
subject matter is represented in ways students cannot understand, the students are at risk 
of failure. In order to diagnose why some representations fail, this project will 
investigate how teachers understand this subject matter, how they select and employ 
representations, and how students understand these representations. Results of this 
study will further the development of teacher performance assessments and add to the 
library of cases available for teacher education. 



Questions/Issues Addressed 

The forms of representations choral music teachers use to communicate the subject 
matter. 

How these representations are alike and different across choral music teachers. 

How a teacher's understanding of the subject matter and of students influences the 
teacher's construction, selection, and use of representations. 



Questions/Issues Remaining 

N/A 
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Rhode Island Literacy Portfolio Presentation Project 



Project Director Mary Fowles 
Project Status Completed 
Educational Level Elementary (Grade 3) 
Subject Matter Literacy (across subjects) 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

This project was an outgrowth of ETS's participation in the design and implementation 
of the Rhode Island statewide writing assessment and the portfolio-based validity study 
for it. The Portfolio Presentation Project was designed to promote effective literacy 
instruction in the classroom, to assess evidence of Literacy learning, and to promote 
assessment as part of the instructional process. Students created portfolios containing 
samples of their work in math, speaking & listening, reading & writing, and other 
content areas. Teams of students presented their portfolios to scorers, and presentations 
were rated on the degree to which they reflected understanding of the process and 
concepts of literacj" development. Ratings were used to assess the effectiveness of the 
Literacy program, not the Literacy levels of individual students. 



Questions/Issues Addressed 

Development of a method for assessing the effectiveness of the literacy program. The 
portfolio presentation method was effective. 

Can instructional assessment enhance programs to promote literacy across subject areas 
at the elementary school level? Yes 

Can elementary students learn to take m.ore active roles in their own learning by 
becoming aware of the learning process, taking responsibility for their own learning, and 
evaluating their own work products? Yes 

Questions/Issues Remaining 

Can such a program be implemented statewide? The state is searching for necessary- 
funding to bring this about. 

Can the program be implemented at other grade levels? Some teachers have expressed 
interest in adapting it for other grade levels. 
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Classroom-Based Documentation of Literacy and Science Development 
in Programs for Urban Minority Children 

Project Directors Jacqueline Jones & Ann Marie Senior 
Project Status Current 

Educational Level Elementary (early childhood) 
Subject Matter Reading, Writing, Science 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

This project is a subcontract of the Trenton Early Childhood Transition Demonstration 
Project, which provides a comprehensive approach to providing educational and support 
services to at-risk children and their families. The Transition Project is designed to 
demonstrate how the gains associated with pre-school programs such as Head Start can 
be sustained and enhanced through implementation of developmentally appropriate 
educational practices across the kindergarten and primary years. The purpose of ETS's 
subcontract is to broaden the base of assessment strategies in early childhood education 
at both the classroom and systemic levels. ETS researchers will observe in classrooms 
and help teachers develop their own classroom-based alternative assessments. 
Researchers will meet with administrators, school board personnel, and the National 
Transition Research Team to investigate the system-wide implications of the new 
documentation procedures. Consultation and workshops for staff training will be 
provided. 



Questions/Issues Addressed 

Development of classroom-based alternative assessment procedures in literacy and 
science for the early elementary grades. 

Investigation and articulation of the system-wide implications of the use of alternate 
assessment procedures. 

Teacher training in the development of alternative assessments and in understanding the 
physical, cognitive, and affective growth and development of children. 



Questions/Issues Remaining 

N/A 
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Interactive Videodisc Applications to Mathematics Education 

Project Director Isaac Bejar 

Project Status Completed (prior to 1985) 

Educational Level Elementary 

Subject Matter Math 

Focus Instruction 

Type of Technology Used Videodisc 
Performance Assessment? No 
Teacher Training Component? No 



Description 

Fraction problems were embedded in the story of a space crew on a rescue mission, 
presented on videodisc. In the field test student reactions were positive, but some 
students did not benefit as much as had been expected from the built-in instructional 
sequences. 

Questions/Issues Addressed 

Evaluate the limitations of videodisc-computer technology for classroom use. 

Assess the economic feasibility of producing a pedagogicaUy and artistically sound 
videodisc. [Unresolved-production was expensive] 

Questions/Issues Remaining 

Specific to this project : Can students' problems with fractions be diagnosed more 
precisely, so that a more effective instructional sequence can be designed? Similar 
problem being addressed in Bennett's AI projects. 

Larger issues : Can high quality videodisc-computer programs be produced within 
reasonable budgets? Still an issue in all technology-based efforts. 

Can results from psychological and educational research be distilled and integrated into 
the design of videodisc-based instruction? Being addressed in numerous projects, with 
respect to technology in general, rather than videodisc applications in particular. 
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Assessment in Mathematics in Monmouth Junction School 

Project Director Ted Chittenden 
Project Status Current 
Educational Level Elementary 
Subject Matter Math 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

ETS staff are providing consultation to teachers on the design and evaluation of 
ended and performance-based assessment tasks for mathematics. 

Questions/Issues Addressed 

Design of assessment tasks appropriate for the district's mathematics program. 



Questions/Issues Remaining 

N/A 
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Documenting and Understanding Young Children's Science Learning 

Project Director Ted Chittenden 
Project Status Current 
Educational Level Elementary 
Subject Matter Science 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

To enhance the teaching of science in the primary grades, programs of documentation 
will be developed, built around classroom observations, records of children's language, 
and samples of children's work such as writings, drawings, and constructions. 

Questions/Issues Addressed 

Can naturalistic methods of instruction and assessment (based on those developed for 
emergent reading by Chittenden, et al.) be used to enhance science learning in grades 
K-3? 

Can appropriate materials be developed to promote teachers' abilities to observe, record, 
and understand young children's science learning? 

Can guidelines and illustrative materials suitable for introducing the concept into leacher 
enhancement and teacher education programs be developed? 

Questions/Issues Remaining 

N/A 
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Alternative Assessment of Science and Mathematics 
for Early Childhood Urban Minority Students 

Project Director Jacqueline Jones 
Project Status Current 
Educational Level Elementary 
Subject Matter Science, Mathematics 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 

Description 

This three-year project is an outgrowth of "Alternative Special Education Assessment for 
Urban Minority Children," with funding from a different source. This project will involve 
five schools in School District 9 in New York City and will focus on science and math. 
The ETS effort is part of Project CUE (Creating Urban Excellence), which seeks to 
develop a science and mathematics program in District 9 and to identify students with 
exceptional talent in these areas. ETS researchers wUl assist teachers in the 
development of performance assessments in science and math and wUl conduct classroom 
observations and weekly meetings with designated groups of teachers, to define types of 
performance data to be collected as well as to develop parameters for the assessment of 
work samples. During the course of the project, a series of survey instruments wUl be 
administered to teachers to evaluate changes in the fundamental ways in which they view 
student ability in science and math and to note any increase in teachers' substantive 
knowledge in these areas. 

Questions/Issues Addressed 

Development of performance assessments in elementary science and math. 

Documentation of changes in teachers' perceptions of student ability in science and math 
as a result of implementation of performance assessment techniques. 

Documentation of gains in teachers' substantive knowledge of science and math as a 
result of implementation of performance assessment techniques. 

Use of performance assessments to identify chUdren with exceptional abUity in science 
and math. 

Questions/Issues Remaining 

N/A 
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Authentic Assessment for Multiple Users 

Project Director Margaret Jorgensen 
Project Status Current 
Educational Level Elementary 
Subject Matter Math, Science 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

ETS staff are working with math and science teachers in grades 3-6 in six school syste • s 
to develop a common rationale and common goals and standards across grades and 
subjects for performance assessment activities that will yield aggregatable data. Goals 
emphasize higher order thinking and metacognitive skills, to be gained and demonstrated 
in the context of the content of the curriculum. 



Questions/Issues Addressed 

Development of goals, standards, and documentation strategies that are consistent across 
grade levels and content areas. 

Questions/Issues Remaining 

N/A 
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Pacific Telesis Foundation "Education for the Future" Program 



Project Director Margaret Jorgensen 
Project Status Canceled in 1992 
Educational Level Elementary, Middle 
Subject Matter Math, Science 
Focus Instructional Assessment 
Type of Technology Used Videotape 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

ETS provided workshops and ongoing support for teachers in the development of 
performance-based assessment/instruction materials. The program targeted at-risk 
students in two California schools. 

Questions/Issues Addressed 

Ways to get teachers to deal with the "Big Ideas" of higher order thinkirtg skills and 
metacognitive processes. The message must be presented in powerful ways: Hands-on 
experience and videotapes of students engaging in and demonstrating metacognitive 
activities are convincing. 

Ways of promoting equity in performance assessments. 1) Relatively short performance 
activities (limited to two class periods, for example) reduce the opportunities for some 
students to take advantage of resources (older siblings, interested parents, materials in 
the home) available to them but not to others; in general, the more extended the 
activity, the greater the chance for lack of equity. 2) Allowing the student a choice of 
response modalities reduces the possibility that the prescribed method of demonstration 
will be punitive to some children. 

Are there some general features of the task of designing performance assessments that 
all teachers learning this approach need to understand? 1) The difference between the 
kind of informal, day-to-day appraisals good teachers routinely carry out and the 
systematic observation to inform instruction, which is the nature of performance 
assessment. 2) The role of simplicity and sampling in performance assessment, in 
preference to extended, complicated projects. 

NOTE: See the description of "Reflections of Learning " Training Seminars for 
explanation of the ways in which these ideas are being put to use. 



Questions/Issues Remaining 

None 



Maryland Education Project, Assessment Measures Task Force 

Project Directors Mary Fowles and Gita Wilder 
Project Status Completed 
Educational Level Elementary, Middle 
Subject Matter Waiting, Math 
Focus Instruction 

Type of Technology Used Computers 
Performance Assessment? Yes 
Teacher Training Component? No 

Description 

ETS worked with the Assessment Measures Task Force of the Maryland Education 
Project to develop a framework for assessing the progress made by schools, teachers, and 
students in integrating computers into instruction. 

Questions/Issues Addressed 

How does the introduction of computers into schools affect the learning process for 
students, the instructional practices of teachers, the classroom process for both teachers 
and students, and the organizational context of the schools? How can these effects be 
judged? A framework was developed that delineates levels of progress in integrating 
computers for instruction in these areas: motivation; use of word processing in the 
writing process; production and performance for purposes of communication; 
integration of reading and writing; gathering, integrating, analyzing, and using 
information; solving complex problems; and adapting instruction to the needs of 
individual students. Within each area, indicators of progress were established for five 
levels of progress in teacher and student attitudes, behaviors, and skills related to the 
use of technology as a teaching and leamuig tool. 

As an example of the kind of information teachers brought to the task force 
deliberations, the project states that the computer contributed to the writing process by 
making it easy for students to revise drafts and by creating products that increased both 
their pride in their ovm' writing and their interest in reading the writing of others. 

Questions/Issues Remaining 

The framework is a described as a working document. Task Force members intend that 
those who use it should provide feedback that will contribute to its continual 
development. 
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Design of an Integrated, Self-Improving Assessment System for California Schools 

Project Director Karen Sheingold 
Project Status Current 

Educational Level Flementaiy, Middle, Secondary 

Subject Matter Ma ill, Science, Language, Social Studies 

Focus Assessment 

Type of Technology Used None 

Performance Assessment? Yes 

Teacher Training Component? No 



Description 

A multi-year research and development project >j design, test, and validate elements of 
the California Assessment Program. 



Questions/Issues Addressed 

Design of curriculum-embedded assessments in each of the subject areas at all grade 
levels. 

Design of criteria for an organic portfolio to be used statewide. 

Testing the viability of these designs for instructional and assessment purposes. 

Design of learning/assessment tasks that are engaging and revealing for all students, to 
assure equity and access. 

Assuring validity of new assessments 

Assuring reliability and generalizability of new assessments. 



Questions/Issues Remaining 

N/A 
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Implementation of Technology in Schools: Lessons from Eight Sites 

Project Directors Brian Slecher, Gita Wilder 
Project Status Completed 

Educational Level Elementary, Middle, Secondary 
Subject Matter Many 
Focus N/A 

Technology Used Computers 
Performance Assessment? N/A 
Teacher Training Component? No 

Description 

This project used a case study approach to examine the factors affecting implementation 
of computer technology in schools. The directors were cautious about making 
generalizations from such a small and diverse sample, and they noted that future studies 
of implementation should be longitudinal, because the implementation process takes 
several years to complete. However, the information they assembled and the conclusions 
they drew will be useful to developers of technology-based programs. 

Questions/Issues Addressed 

Importance of the implementation process to the ultimate effectiveness of a technology- 
based program. The implementation process is "an important (if not the most 
important) element in technological innovation and that this pro- ss is both complex 
and lengthy, lasting many years in the sites we visited." 

Factors that influence the implementation of technology-based programs in public school 
settings. The two most important factors in successful implementation were 

1) a close match between the technology and the existing curriculum and 
instructional patterns, and 

2) strong and continuing administrative support and leadership. 

The change required in teachers' roles and responsibilities was a significant barrier to 
successful implementation, although this barrier was overcome in some schools because 
of strong, committed leadership that fostered a climate for change. 

(See ETS RM-91-4 for full discussion.) 



Questions/Issues Remaining 

Study of the implementation process at the classroom level, to determine the effects of 
teacher and classroom-level variables on the implementation process. 
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PACKETS^ 



Project Director Nancy Katims 
Project Status Current 

Educational I^evei Elementary, Middle, Secondary 

Subject Matter Ail 

Focus Instructional Assessment 

Type of Teclinoiogy Used None 

Performance Assessment? Yes 

Tejicher Training Component? No 



Description 



A series of performance-assessment materials for use by classroom teachers. Each set 
focuses on a particular subject and level. The first (and only, so far) is middle school 
math, using materials developed by Dick Lesh. Materials for all PACKETS will be 
produced as part of other ETS performance-assessment projects. 



Questions/Issues Addressed 

None. All research questions are addressed in the course of the projects in which the 
materials are originally developed. 



Questions/Issues Remaining 

N/A 
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"Reflections of Learning" Training Seminars 

Project Director Margaret Jorgensen 
Project Status Current 

Educational Level Elementary, Middle, Secondary 

Subject Matter All 

Focus Instructional Assessment 

Type of Technology Used Videotape 

Performance Assessment? Yes 

Teacher Training Component? Yes 

Description 

Training seminars are offered for classroom teachers in all subjects, to teach them how 
to develop and use performance assessment activities to promote learning of higher 
order thinking and metacognitive skills in the context of their disciplines. Standards 
developed by professional organizations in the content fields are used to define the 
thinking skills to be taught {e.g., problem solving, reasoning, reflecting on the learning 
process, self-correcting, using different ways to communicate ideas, predicting). 

Questions/Issues Addressed 

N/A, but see the description of the Pacific Telesis project for a description of some of 
the principles applied to the seminars. 

Questions/Issues Remaining 



N/A 



Videodisc Technology and Sign Language for Improving Reading Skills of Deaf Middle School 
Students 

Project Director Eric Hansen 
Project Status Current 
Educational Level Middle 
Subject Matter Reading 
Focus Instruction 

Type of Technology Used Computer, Interactive videodisc 
Performance Assessment? No 
Teacher Training Component? No 



Description 

This project explores the challenges deaf students face in comprehending written English 
and examines the assistance that may be afforded by signed versions of text presented 
via an interactive videodisc system capable of presenting reading comprehension 
passages and questions in either American Sign Language (ASL) or English-based sign. 
In Phase 1, each participant in the study will be presented with reading passages and 
accompanying questions in six different forms: English test only, ASL only, English- 
based sign only, English test plus optional ASL, English text plus optional English-based 
sign, and English test plus both signed options. In Phase 2, participants will use 
additional system functions and tools such as corrective feedback, a dictionary, and 
diagnostic probe questions. 



Questions/Issues Addressed 

Phase 1: Issues of performance and preference for a particular language version will be 
studies in light of content variables and student variables. 

Phase 2: The nature of the difficulties deaf students face in accessing the meaning of 
written English, e.g., how deaf students with different backgrouncs try to handle difficult 
inferencing problems; to what extent deaf students have trouble with particular 
vocabulary versus higher-level constructions. 



Questions/Issues Remaining 



N/A 



Ii.''side Story: Dateline Brazil 

Project Director Nancy Katims 

Project Status Completed 

Educational Level Middle School 

Subject Matter Language Arts/Critical Thinking Skills 

Focus Instruction 

Type of Technology Used Computer (CAI) 
Performance Assessment? No 
Teacher Training Coiuponent? No 

Description 

A computer-assisted instruction unit in critical thinking skills was developed for use in 
language arts classes. The unit combines computer activities with classroom activities. 
Assessment is done by the teacher through activities within the unit. 

Questions/Issues Addressed 

Can the computer be used to present an interactive critical-thinking-skills unit 
appropriate for middle school students? Yes 

Questions/Issues Remaining 

How well can such a unit be incorporated into existing language arts programs? 

What kinds of teacher training are necessary for it to be implemented effectively? 

To what extent do students acquire transferrable critical thinking skUls as a result of 
working through the unit? 
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Evaluation of a Mathematics Videodisc Project 

Project Director John V. Moe 
Project Status Completed 1992 
Educational Level Middle School 
Subject Matter Math 
Focus Instructional Assessment 
Type of Technology Used Videodisc 
Performance Assessment? No 
Teacher Training Component? No 

Description 

Two mathematics videodisc lessons were developed as part of a project to interest 
minorities and girls in mathematics and to assist teachers as they teach mathematics. 
Only one of the two was used in final tryouts, which took place in three schools with 
nearly 100% minority enrollments, in three different cities. An important feature of the 
project was the use of an electronic student response system to monitor student 
responses to questions presented as instruction occurred. 

Questions/Issues Addressed 

How effective were the lessons in terms of student achievement? They were effective for 
virtually all students and more effective for girls than for boys. Girls, who scored lower 
than boys on the pretest, made greater gains than boys, although their posttest scores 
did not quite reach the level of boys' posttest scores. 

Other findings: 

Using an electronic student response system encourages shy students and students who 
have had little previous experience with the topic to participate actively in the lesson. 

Blending technological applicaf 'ons with more traditional approaches (eg., teacher 
explanations, students working at the blackboard) is likely to be more effective than 
using either approach alone. 

[Technical recommendations specific to this application of videodisc technology were 
also included in the project report.] 

Questions/Issues Remaining 

Further exploration of the effects of frequent, low threat student/teacher and 
student/student interactions on the learning process for groups of students. 

Expansion of this technique to other areas of the curriculum. 

Incorporating recent theories of cognition into the development of other lessons using 
the videodisc technique. 



Research and Development in Mathematics using "Authentic Assessments" 

Project Director Dick Lesh 
Project Status Current 
Educational Level Middle School 
Subject Matter Math 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 

Description 

Development of math activities that combine assessment and instruction. Math 
problems are presented in the context of realistic newspaper anicles that yield three 
types of questions: 

Test-like questions that deal with facts and skills; 

Homework-like problems that are somewhat more complex; 

"Model-eliciting" projects that produce results that become the basis for a new 
article. These may take up to 50 minutes to solve and are often undertaken as 
group activities. 

Questions/Issues Addressed 

Can "authentic assessments" such as that described above be created? Yes 

How effective are such assessments in increasing students' understanding of and ability 
to use mathematical principles? 

In the "model-eliciting" problems, how can reliability be assured? 

How can teachers and test developers learn to create such assessments? 

Will the use of such assessments help to identify more students with high math ability? 

Can assessments be designed that are targeted toward groups in which math 
participation is currently low? 

Questions/Issues Remaining 

N/A 
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Project GIMS (Georgia Innovations in Math and Science, an NSF statewide systemic initiative) 



Project Director Margaret Jorgensen 
Project Status Current 
Educational Level Middle School 
Subject Matter Math, Science 
Focus Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

ETS is one participant in the NSF-sponsored "statewide systemic initiative program for 
Georgia, designed to radically change instruction in science and math. The University of 
Georgia administers the program; the ETS Atlanta Office is responsible for designing 
assessnients. 



Questions/Issues Addressed 

Subcontract is essentially for development. As the project progresses, research issues 
may emerge. 



Questions/Issues Remaining 

N/A 
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SEPIA (Science Education through Portfolio Instruction and Assessment) 

Project Director Drew Gitomer 
Project Status Current 
Educational Level Middle School 
Subject Matter Science 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

The project is designed to foster the construction of scientific meaning in students and to 
encourage conceptual development consistent with the growth of scientific knowledge. 
Students will create portfolios containing collections of their work that reflects the 
processes of scientific thinking, such as students' explanations of scientific phenomena, 
student-created models of systems, reflections on science activities, and self-evaluations 
of progress. Teachers will develop assessment techniques that reveal more about 
students' understanding of science topics than do traditional tests. 

Questions/Issues Addressed 

How effective is the portfolio-based approach in fostering a conceptual-change model of 
teaching? 

How do students adapt to their changed roles, as they are asked to take more 
responsibility for their learning and to take more control in assessing their own work? 

What instructional and curriculum guidelines will effectively, assist teachers in conducting 
classroom activities and selecting and sequencing instructional tasks? 

What teacher in-service programs are needed in order to implement conceptual-change 
teaching reforms? 

What strategies and evaluation criteria can be developed that will assess students' 
knowledge bases in science and reflect the principles of scientific thinking? 

What do alternative assessments designed by teachers reveal about students' 
understanding? How can that knowledge be used to inform further instruction? 

How can alternative assessment strategies like portfolios be used to communicate 
learnmg outcomes to administrators and parents? 

Questions/Issues Remaining 

N'/A 




Middle School Science Project 



Project Director Brian Gong 
Project Status Current 
Educational Level Middle School 
Subject Matter Science 
Focus Instructional Assessment 
Type of Technology Used Computers 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

Development and implementation of a program of instructional assessment to enhance 
the learning- of science at the middle school level. 

Questions/Issues Addressed 

How can the development of an adaptive teaching/learning environment be facilitated? 
What kind of teacher training is needed to implement such a program? 
How can the school organization promote this type of adaptive teaching/learning? 
Can science units be developed that 

integrate assessment with instruction? Yes 

integrate technology and its uses with other aspects of science? Yes 
motivate students by focusing on everyday, real-world problems? Yes 

Can the learning of science be integrated with other subjects? Yes 

What kinds of mental models do students at this level use to understand and explain 
scientific phenomena? How do different forms of instruction contribute to the 
development of different kinds of mental models? 

How do teachers use assessment information to guide practice? 



Questions/Issues Remaining 

N/A 



Performance Assessment as a Tool for Enhancing Learning in Middle School Science 



Project Director John Frederiksen 
Project Status Current 
Educational Level Middle, Secondar}' 
Subject Matter Science 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

The project will explore the use of student projects and extended performance tasks 
("inquiry-based science") that enhance the teaching and learning of science, that meet 
psychometric standards for validity, fairness and reliability, and that provide information 
necessary for public accountability. 

Questions/Issues Addressed 

Can principles be developed for constructing and scoring performance tasks? 

What qualities of performance should be assessed, and how should they be presented? 

What is the best way to construct systems ("rubrics") for scoring complex performances? 
How useful for various assessment puqjoses are the analytical and the holistic models? 

Can scoring rubrics be designed :o communicate to students, through instructional 
feedback, the goals of scientific inquiry? 

How can teachers be trained to accurately and reliably score performance tasks? Can 
consensus moderation groups be used to maintain the accuracy of scoring? 

How important is standardization of performance tasks and conditions of 
administration? 

Do students show improvement in their quality of performance over a sequence of 
assessment tasks throughout a semester or year? Is improvement generaUy noticeable, 
or is it restricted to particular performance tasks? 

Does the introduction of a performance-assessment system result in beneficial changes in 
teaching practice? What additional teacher training is required? 

What instructional models will develop knowledge and skill in scientific inquiry in less 
advantaged populations? 



Questions/Issues Remaining 

N/A 
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Evaluation of Hunterdon Central High School Prototype Classrooms 

Project Director Lori Morris 

Project Status Current 

Educational Level Secondary 

Subject Matter Mathematics, Science, Industrial arts 

Focus Instruction 

Type of Technology Used Computers, telecommunications, laser and CD players, projection 

capability 
Performance Assessment? No 

Teacher Training Component? No (but see description below) 



Description 

The school system is establishing three prototype classrooms (a biochemistry lab, a 
physics/electronics lab, and an applied technology lab), using advanced technology. Each 
room will include a teacher/presentation area equipped with laser disc and CD players, 
telecommunications capacity, and projection capability for video and computer-generated 
material. The biochemistry lab will have 12 computer stations, each connected with the 
school's library/media center/communication center as well as to external databases. In 
addition, the school has also created a computer lab for the Mathematics Department. 
The purpose of this project is to evaluate what teachers need in the way of training and 
curriculum development. The project director will also help the school district seek 
funding for these activities. 

Questions/Issues Addressed 

What training do teachers need in order to be able to make effective use of the high 
technology classrooms? 

What curriculum development activities should be undertaken? 



Questions/Issues Remaining 

N/A 
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Video Portfolio Assessment 



Project Director John Frederiksen 
Project Status Current 
Educational Level Secondary 
Subject Matter Math 
Focus Assessment 

Type of Technology Used Videotapes 
Performance Assessment? Yes 
Teacher Training Component? Yes 

Description 

This project explores the feasibility of using classroom videotapes as a source of data for 
an assessment system for certifying masterful teachers. The goal of the project is to 
create a performance assessment of accomplished teaching that will make a direct and 
positive contribution to the professional development of teachers. The project focuses 
on math teachers at the secondary level; the principles developed through the poject will 
be applicable to other subjects and levels. 

Questions/Issues Addressed 

What concepts do accomplished teachers use in evaluating teaching? The project 
identified four criteria: Pedagogy, Climate, Mathematical Thinking (or a corresponding 
content specific goal for other subjects, and Management. 

Can teachers develop common perceptions of teaching based on a shared set of concepts 
used as an interpretive framework? Yes. 

Can teachers integrate evaluations based upon separate concepts in making overall 
judgments of teaching expertise? Judgments can be made for each criterion of teaching 
expertise; however, because the weighting of vf iteria and the integration of video 
portfolio scores with other information are beyond the scope of the project, no overall 
scores were assigned to the portfolios. 

Can the contents of video portfolios be scored with reliability? Yes. Further research is 
being done to create a model of the scoring process and to compare it with the scoring 
models used in the classroom observation component of The Praxis Series: Professional 
Assessments for Beginning Teachers™. 

What is the best way to facilitate development of a sociaUy shared understanding of 
assessment concepts among teachers who develop or score video portfolios? One good 
way is through video clubs, in which teachers meet to view and reflect upon their own 
and others' videotapes and to discuss, the criteria on which the portfolios will be scored. 
Further research on video clubs is under way. 

Questions/Issues Remaining 

N/A 




CIESE Projects — Evaluation of Computer Integration in Math Education 

Project Director Lori Morris 
Project Status Completed 1992 
Educational Level Secondary 
Subject Matter Math 
Focus Instruction 

Type of Technology Used Computers 
Performance Assessment? No 
Teacher Training Component? No 

Description 

The Center for Improved Engineering and Science Education (CIESE was established by 
Stevens Institute of Technology to help integrate computer-based instruction in 
mathematics at the pre-college level. This project was part of the evaluation of the 
effectiveness of the CIESE project at the secondary level in five demographically diverse 
school districts. 

Questions/Issues Addressed 

Barriers to integration. Primary barriers are time (to preview software and prepare 
lesson plans) and money (to piu-chase hardware, software, disks, and paper). CIESE's 
formal presentations to school boards and informal conversations with administrators 
played an integral role in resolving these issues. 

Impact on teachers. Computer integration encouraged teachers to self-reflect on their 
teaching styles and pedagogies. Teacher's role changed from authoritarian to coach. 
Teachers of higher academic level students tend to use the computer to encourage 
higher order thinking skills through the discover method of learning (rather than using 
the time saved to cover a wider range of skills). 

Changes in student attitudes. Students gained autonomy in their pursuit of 
understanding of math, being able to make mistakes privately at the computer rather 
than in front of the class. Discussion of math among students increased. In lower 
ability classes, students' self-esteem and attitudes toward math improved and discipline 
problems decreased. 

Effects on the social structure of the schools. The process of integration encouraged 
collaborative relations among teachers, and, in some cases, between administrators and 
teachers. 

Responsiveness of the CIESE project to the needs of teachers. CIESE staffs 
nondirective approach empowered teachers, providing them with confidence to 
independently develop their own criteria for judging software. CIESE workshops were 
considered successful by teachers because time was provided for teachers to reach their 
goals; workshops were flexibie enough for teachers to work informally in groups or 
alone; teachers had the opportunity to interact with math teachers outside their schools; 
teachers felt they were treated like professionals. (continued on next page) 
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CIESE Projects (continued) 

Other findings. Advantages of the computer cited by teachers include enhanced qualitj- 
and quantity of visual examples and ability to cover more material in a shorter time. 
Evaluation of student achievement outcomes (by Stevens researchers) showed that 
students made gains that were maintained over time. 



Questions/Issues Remaining 

None 
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Use of Artificial Intelligence to Score Constructed-Response Items in Algebra 

Project Director Randy Bennett 
Project Status Current 
Educational Level Secondary 
Subject Matter Algebra 
Focus Instructional Assessment 

Type of Technology Used Computer, Expert System, Automated Scoring 
Performance Assessment? No 
Teacher Training Component? No 

Description 

Development of interactive math assessments in which students work on problems 
similar to those encountered in academic work or employment. Students' solutions are 
scored automatically. Partial credit is given for varying degrees of correctness. 
Diagnostic information about errors is provided. 

Questions/Issues Addressed 

Can constructed responses to math items be scored automatically, giving partial credit 
for partially correct responses, and providing diagnostic information about errors? Yes 

Can an automatic scoring system diagnose and evaluate the process used to arrive at an 
answer? Yes 

How does the effectiveness of the scoring system vary across four different types of 
open-ended questions, representing four degrees of structure in presenting the problem 
and four levels of constraint in the response? 

How applicable are new, cognitively-driven measurement models to such interactive 
assessments? 

Can the system be extenc' d to other content classes and to other difficulty levels within 
the algebra domain? 

Can the scoring rubric be refined to reduce discrepancies between automatic and human 
scorers? 

Do these constructed-response item types measure a different dimension from that 
measured by traditional (multiple-choice) items? 

How can information be aggregated across itens to detect and describe instructionally 
meaningful patterns in the scores and diagnostic information? 

Questions/Issues Remaining 

Can such a system be integrated into classroom instruction? 
Does the system enhance the learning process in algebra? 
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A Practice and Feedback System for AP Computer Science 

Project Director Randy Bennett 
Project Status Current 
Educational Level Secondary 
Subject Matter Computer Science 
Focus Instructional Assessment 

Type of Technology Used Computer, Expert System, Automated Scoring 
Performance Assessment? ^(M 
Teacher Training Component? No 

Description 

The Advanced Placement Computer Science Practice System was developed to explore 
issues relating to classroom instructional assessment, to the use of complex constructed- 
response tasks in large-scale testing, and to the integration of the two. Items included in 
the system emphasize important data structures taught in the APCS curriculum. After a 
problem is selected, the student is transferred to the editing mode, in which the student 
can write, compile, and run a solution, and then request a semantic analysis and grading 
of the solution. The system provides a score on the APCS scale and diagnostic 
comments about errors. 

Questions/Issues Addressed 

Can responses to performance tasks in computer science be analyzed automatically? Yes 

Can a system with the following features be designed? Yes 
runs on classroom computers 
analyzes responses in real time 
allows student to interact with it directly 
can be modified to add new problems 

How can knowledge bases fo' diagnostic analysis be constructed? By reviewing. problem 
solutions from ETS files, identifying approaches students used and errors they made, 
and encoding the correct and incorrect elements of the solutions. Example Compiler, a 
development tool, was developed to facilitate the process. 

Addition of other classes of problems ("class" = group of problems all having the same 
underlying data structure) representing other data structures included in the AP 
curriculum. 

Questions/Issues Remaining 

Evaluation of effectiveness of classes of problems in helping students learn to recognize 
the underlying structure of a problem. 

Evaluation of operational classroom use of the system in AP courses. 
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Sign Language and Videodisc for Tes* Preparation Materials for Deaf Secondary School 
Students 

Project Director Eric Hansen 
Project Status Current 
Educational Level Secondary 
Subject Matter Reading 
Focxis Instruction 

Type of Technology Used Computer, Interactive videodisc 
Performance Assessment? No 
Teacher Training Component? No 

Description 

Development and evaluation of a computer-based test preparation system using sign 
language for deaf secondary school students preparii?g for reading comprehension tests. 
The system will consist of a pretest, an instruction component, and a posttest. In the 
instruction component, the student will be able to request American Sign Language 
translations of virtually all text. 

Questions/Issues Addressed 

Effectiveness of the system in preparing students for reading comprehension tests. 

Questions/Issues Remaining 

N/A 
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Systems Thinking and Curriculum Innovation Network (STACt^ 

Project Director Ellen Mandinach 
Project Status Current 
Educational Level p|l|Jdpe, Secondary 
Subject Matter Science, Math, Social Studies, Humanities 
Focus Instruction 

Type of Technology Used Computers and Simulation-Modeling Software 
Performance Assessment? |||s 
Teacher Training Component? Yes 

Description 

A multi-year research effort examining the cognitive impact of the use of a systems 
thinking approach and simulation-modeling software to supplement and enhance 
instruction in a variety of content areas. The project studies the effects of systems 
thinking on acquisition of content-specific knowledge and general problem-solving skills, 
the effectS'on teacher pecfofmaitce aad'dassrooax prodesses, and the e^ect on 
OganiSSatiOH^ $tn»i^.or6. and IttijcttCKofeg. 

Questions/Issues Addressed 

Can secondary school students learn the principles of systems thinking and apply them in 
their courses? Yes 

Do students acquire problem solving skills through the use of systems thinking and 
simulation modeling? Yes 

In contrast to students in traditional courses, do systems-thinking students evidence 
greater conceptual understanding of topics being studied? Yes 

How does use of the systems approach affect students' acquisition of content knowledge? 

What are the effects of the use of various strategies and methods for integrating 
modeling activities into the curriculum? 

What mathematical skills are prerequisite to the effective use of modeling activities? 

What \s the effect of cognitive development ont students' ability to benefit from the 
systemsrappmadh.? 

Do analytic reasoning skills have a cumulative effect as students progress through several 
years of systems thinking courses? 

How does the role of the teacher change when a systems approach is implemented? 
Teacher becomes a facilitator who works interactively with students. 

What kind of training can best prepare teachers to use the systems approach? 
Questions/Issues Remaining N/A 
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Pacesetter 



Project Director Ernest Kimmel 

Project Statxis Current 

Educational Level Secondary 

Subject. Matter English, History, Science, Math 

Focus Instructional Assessment 

Type of Technology Used None 

Performance Assessment? Yes 

Teacher Training Component? Yes 

Description 

"Pacesetter, a major college Board initiative, is an array of secondary school syllabi and 
related assessments, supported by professional development opportunities for teachers. 
It is designed to develop challenging standards, raise expectations and improve 
performance for all American students, and to ensure that educators are at the forefront 
of defining and delivering that effort. Developed in cooperation with leading 
professional associations in the academic disciplines and educational practitioners at all 
levels, its initial offering in mathematics is projected for 1993, followed by offerings in 
English, world history, science, and another subject to be determined." 

Questions/Issues Addressed 

Development and evaluation of the following components: 

The syllabus, describing course content and anticipated learning outcomes. 

Teacher training and support activities. 

Classroom assessments intended to permit teachers to monitor and shape instruction, as 
well as to provide feedback to students. 

End-of-course achievement tests, which wiU include both multiple-choice and 
performance-based sections. 

A system for scoring the end-of-course achievement tests on a state, regional, or local 
level, including the training of teachers who score them. 

Questions/Issues Remaining 

N/A 
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Arts PROPEL 



Project Director Drew Gitomer 
Project Status Completed 1992 
Educational Level Mi<M^, Secondary' 
Subject Matter Writing, Art, Music 
Focus Instructional Assessment 
Type of Technology Used None 
Performance Assessment? Yes 
Teacher Training Component? Yes 



Description 

The goal of the project was to develop powerful modes of encouraging and assessing 
artistic learning in a wide range of students, not just those recognized as artistically 
talented. The project focused on artistic development in imaginative writing, music, and 
visual art. The project developed a model that employs "domain projects," which consist 
of sequences of instruction about a central problem or project. Using portfolios, 
students selected, saved, and reflected upon the diverse body of material that underlay 
their final productions or performances. 



Questions/Issues Addressed 

Can assessment and instruction be integrated as interactive and iterative components of 
the learning process? Yes 

Can students learn to look critically at their own work and the work of their peers and to 
reflect upon the learning and creating processes? Yes 

Can teachers adapt to new roles as facilitators and moderators? Yes 

Does this approach improve student learning? Yes 

Questions/Issues Remaining 

Can this approach be adapted to other subject areas? An experimental project in 
science (SEPIA) is under way. 
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Issues and Technical Procedures in Rating Portfolios 

Project Directors Carol Myford & Robert J. Mislevy 

Project Status Current 

Educational Level Secondary 

Subject Matter AP Studio Art 

Focus Assessment 

Type of Technology Used None 

Performance Assessment? Yes 

Teacher Training Component? No 



Description 

Focusing on the AP Studio Art program, this project examines issues related to portfolio 
assessment and performance assessment from both statistical and naturalistic 
perspectives. From the i;tatistical perspective, the project is using data from the A? 
Studio Art program to gain insights into important generalizability problems and to build 
expertise with analytical tools for such data. The naturalistic perspective provides 
insights into the processes underlying the patterns detected by statistical analysis. The 
issues being addressed are those that arise in many performance assessment settings, and 
the technical procedures used can be applied in a variety of contexts. The project can 
serve as a model for other programs that seek to monitor the effectiveness of rating 
procedures and to establish quality control over those procedures. 

Questions/Issues Addressed 

DefLnLng rating standards. 

Assuring consistency in ratings (reliability). 

Assuring the construct validity of the AP Studio Art portfolios; examination of possible 
differences between the values and perspectives of secondary school art teachers and 
those of art teachers at universities and art schools. 



Questions/Issues Remaining 

N/A 
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